Automatic registration of intra-modality medical volume images using affine transformation

ABSTRACT

A current diagnostic image and an archived diagnostic image of a common region of patient are loaded into a first memory ( 14 ) and a second memory ( 18 ). The diagnostic images are converted into feature images ( 24 ), scaled ( 40 ), and normalized ( 42 ). An affine transform determining processor ( 50 ) generates an affine transform representative of the error between the current and archived images. A transform operator ( 90 ) operates on one of the diagnostic images in accordance with the affine transform to bring the two images into registration. A display processor ( 104 ) displays corresponding pairs of slices of the registered first and second images on a monitor ( 22 ). A stepping processor ( 102 ) causes the displayed slice pairs of the registered images to be stepped together in coordination.

The present invention relates to the diagnostic imaging arts. It findsparticular application in conjunction with CT oncological studies of thelungs and will be described with particular reference thereto. However,it is to be appreciated that the present invention is applicable to awide range of diagnostic imaging modalities and to the study of avariety of organs for a variety of reasons.

When a patient is undergoing treatment for lung cancer, the lungs areperiodically re-inspected, such as with a CT scanner. The oncologistcompares the current images from the CT scanner with the images of thesubject that were taken at an earlier time. Based on this comparison,the oncologist can determine the rate of progression of the disease orwhether it is in remission.

Typically, the oncologist calls up both the current image data set andthe prior image data set. The oncologist displays a transverse slicethrough the lungs of one of the image sets at the top of the videomonitor and a transverse slice of the patient's lungs from the otherdata set at the bottom of the monitor. The oncologist manually stepsthrough the two data sets independently and manually determinescorresponding slices. Often, several thinner slices are fused into athicker slice.

One of the difficulties resides in the matching of corresponding slices.The matching is both time consuming and subjective.

Moreover, the data from the two data sets may have been generated withdifferent imaging parameters, with a different placement of the patientrelative to the center of the scan circle, and the like. The two datasets may have slices of different thickness or skewed at differentangles. The images may be shifted relative to the center of the scancircle. There may be differences in the field of view or scale of theimages. The lungs may have been imaged at a different pulmonary phase.These and other such factors all contribute to the difficulty of manualimage alignment and increase the prospect for subjectivity or humanerror.

Previously, technique have been developed that use fiducials to aligntwo images. That is, fiducials or imageable markers are affixed to thepatient closely adjacent the region of interest so that they are in afixed relationship. These markers are readily aligned in the two images.

However, when the images are taken some duration apart, maybe months,maintaining the fiducials attached to the patient's exterior for thatduration is inconvenient. Moreover, fiducials are used less commonly forinternal organs in the torso because the patient surface moves inrelation to most internal organs with respiration, body position, andthe like.

Images have also been aligned using anatomical markers. That is,characteristic portions of the body around the region of interest, suchas unique locations on the skull or the vertebrae, have been usedinstead of externally applied fiducials. However, distinct anatomicalmarkers are not readily available for lungs and other organs which canmove relative to them. Consequently, most fiducial and anatomical markeralignment techniques have been used in rigid portions of the body, suchas the head inside the rigid skull or along the vertebrae.

The present application contemplates a new and improved automatedregistration technique which overcomes the above-referenced problems andothers.

In accordance with one aspect of the present invention an apparatus fordiagnostic imaging is disclosed. A first memory stores a firstdiagnostic image. A second memory stores a second diagnostic image. Ameans automatically registers the first and second diagnostic imagesfrom the first and second image memories without operator assistance. Ameans for concurrently displays corresponding slices of the first andsecond registered diagnostic images. A means concurrently steps thedisplayed slice to corresponding regions of the first and second images.

In accordance with another aspect of the present invention a method ofdiagnostic imaging is disclosed. A first and a second diagnostic imagesare stored. The first and second diagnostic images are automaticallyregistered without operator assistance. Corresponding slices of thefirst and second registered diagnostic images are concurrentlydisplayed. The displayed slices are concurrently stepped tocorresponding regions of the first and second images.

One advantage of the present invention is that is quick, easy to use,and simple.

Another advantage of the present invention is that it is completelyautomatic.

Another advantage resides in its precision.

Another advantage resides in the coordinated viewing and steppingthrough the images of both scans in a single action.

Yet other advantages reside in the ease of follow-up examinations of thesame patient.

Still further advantages and benefits of the present invention willbecome apparent to those of ordinary skill in the art upon reading andunderstanding the following detailed description of the preferredembodiments.

The invention may take form in various components and arrangements ofcomponents, and in various steps and arrangements of steps. The drawingsare only for purposes of illustrating the preferred embodiments and arenot be construed as limiting the invention.

FIG. 1 is a diagrammatic illustration of an apparatus whichautomatically registers 3D images of a whole body part;

FIG. 2 is an expanded diagrammatic illustration of a part an apparatusof FIG. 1 which automatically registers 3D images of a whole body part;

FIG. 3 is a diagrammatic illustration of the affine transform means ofFIG. 1 and associated parts of the apparatus;

FIG. 4 is an expanded diagrammatic illustration of a part the apparatusof FIG. 1 which automatically registers 3D images of a whole body partand displays them on the monitor.

With reference to FIG. 1, a patient is positioned in a diagnostic imager10, such as a CT scanner, for a follow-up examination. The generateddata is reconstructed by a reconstruction processor 12 and stored in a3D, volumetric image memory 14. Various image enhancement operations asare known in the art are also performed.

Image data from the hospital archive or from another storage medium 16of the same region of the same patient is retrieved and stored in anarchived 3D volumetric image memory 18. Of course, both the current andarchive 3D image memories may be parts of a common storage medium. Aregistering means 20 retrieves the current and archived images andautomatically register the two images for a concurrent display on amonitor 22.

With continuing reference to FIG. 1 and further reference to FIG. 2, asurface image generating means 24 generates surface images from thecurrent archived images. A segmenting means 26 extracts the importantbody parts for the registration, then a feature extraction means 28extracts common features for registration. Looking specifically to lungimages, the segmentation includes a thresholding means 30 thresholds theimages. More specifically to the lung embodiment, the thresholding means30 sets the gray scale value of air in the lungs to one extreme value,e.g., black or white, and all tissue in the image to the other.Specifically to lung images, the feature extraction means 28 includes asurface voxel extracting means 32 which extracts the single layer ofvoxels which lies along the interface between the black and whiteregions. Preferably, the voxels which represent the tissue surfacerather than the air surface are selected. A feature exclusion means 34removes part of the features of the 3D image that is defective orartifacted, if any. Looking specifically to lung images, portions of thelung closely adjacent to the heart often suffer motion artifacts. Tosimplify the remaining operations, the feature exclusion means 32optionally removes a common artifacted region from both imagerepresentations. The voxels are stored in feature or surface imagememories or memory sections 36, 36′.

A scaling means 40 scales the two surface images in accordance withknown differences in scanning parameters. For example, the images areadjusted for differences in the slice thickness of the scans, for thefield of view, for magnification differences, and the like. Anormalizing means 42 normalizes the two surface images. Morespecifically to the preferred embodiment, the extreme points along thez-axis of each image are labeled −1 in one direction and +1 in the otherdirection. Points in between are labeled proportionately. The samescaling is carried out with respect to the x and y-axes. Differentnormalization methods are also available, for example, computing themean or median and standard deviation of the data and setting thecentral point to the mean or median and the length of one is set to thestandard deviation from the mean or median. Of course, other coordinatesystems are contemplated. The two scaled and normalizedthree-dimensional surface images are stored in scaled and normalizedvolume image memories or memory sections 44, 44′.

With continuing reference to FIGS. 1 and 2 and further reference to FIG.3, an affine transform means 50 determines the twelve values of theaffine transform that defines the registration error between the currentand archived scaled and normalized 3D images. Specifically, the affinetransform means 50 determines nine rotational components about threeorthogonal axes and three translational components along the three axes.Optionally, a scaling parameter can also be determined.

The affine transform means 50 includes a selective or non-uniform randomfeature or point reducing means 52. That is, with normal CT resolution,the two surface images are still hundreds of thousands of voxels. Toreduce the computational load, the selective (based on prior knowledge)point reducing means 52 or a non-uniform random point reducing means 52preferably selectively (based on prior knowledge) randomly selects afraction of the points, e.g., 1%, from both images. Because thediagnosing oncologist typically is most concerned with registrationalong the z- or longitudinal axis, the non-uniformity with which thepoints are selected is altered to favor points, which influence thez-registration. In the preferred embodiment, a higher percentage ofpoints are selected along the top and bottom axes of the lungs. Amatching means 54 matches the non-uniform randomly selected points fromthe one surface image with a corresponding point in the other surfaceimage. In the preferred embodiment, a K-D tree matching technique orprocessor 56 is utilized. In each iteration, a non-uniformly randomsample of the points from the current image, to be matched to thecorresponding point in the other surface image, is taken to be matchedto the other set of points from the previous image. For each point inthe sample, a matching point is found using the K-D tree. For eachmatching pair, the coordinates or location of each voxel of the pair andthe surface normal are determined. In the matching operation, if morethan one candidate point has an equally good spatial match to thereference point, the point or voxel with the closest normal is selectedto generate a set of points and normals which are stored in a point andnormal memory means 58.

A bad pair removal means 60 removes matched pairs of points that fail tomeet preselected criteria. In the preferred embodiment, these criteriainclude a physical separation distance greater than a preselectedminimum, badly misaligned normals, and points near to the cut surface,if any. An error estimating means 62 estimates the error between thepoints. The error estimating means 62 calculates a weighted distanceusing the differences in x, y, z, and the normals. Each coordinate isassigned with different weight so that the differences in specific partswill be favored over other parts. Looking specifically to lung images,the differences in the z-coordinate have higher weight than the x andy-coordinates, and higher than the normal differences.

With continuing reference to FIG. 3, an error minimizing or affineregistration means 70 finds an affine transform that minimizes the errorbetween the pairs of points. In the preferred embodiment, the errorminimizing means 70 uses a Levenberg-Marquardt minimization technique orprocessor 72. Using the weighted error estimate and the divergence ofthe weighted error estimate, the error minimizing means finds 70 a setof affine transformation parameters that gives the minimum of error forthe set of sample pairs. Twelve affine transform coordinates andoptionally the scaling correction are loaded into a correction transformmemory 80.

With continuing reference to FIG. 3, a transform processor 82 transformsone of the current and archived scaled and normalized surface imagesfrom memories 44, 44′ in accordance with the determined correctiontransform. The affine transform means 50 repeats the registrationtechnique re-registering the volume images, with one of them subject tothe transform, which was predicted, to transform them into registration.This process generates the twelve affine transform coordinates (andoptionally scaling) of a correction transform to the first correctiontransform. An end criteria determining means 84 determines whether thetwo images have been aligned within acceptable tolerances by monitoringthe correction transforms. In the preferred embodiment, the end criteriadetermining means 84 determines whether the global translation is lessthan a preselected threshold, whether the change in the correctiontransform is below a preselected threshold, or whether a preselectednumber of iterations have been performed. If any of these criteria aremet, a correction transform combining means 86 combines the firstcorrection transform and each of the subsequent correction transforms toprovide a single registration transform which is stored in aregistration transform memory or buffer 88.

With reference to FIG. 1 again and further reference to FIG. 4, atransform operator 90 operates on one of the current and 3D archiveimages, preferably the archived image to register it with the current 3Dvolume image. The transformed archived image is stored in a transformed3D image volume memory 92.

An operator, through a keyboard or other input device 100 controls astepping means 102 which causes a display processor 104 to withdraw anddisplay corresponding slices from the current 3D image memory 14 and thetransformed 3D image memory 92 on a monitor 22. In the preferredembodiment, the transverse slice from the current image memory 14 isdisplayed on the top half of the screen and the corresponding slice ofthe transformed memory 92 is displayed on the bottom half of the screen.Preferably, corresponding orthogonal views are displayed adjacent thelarger longitudinal slice. The operator input device 100 also controls aslice fusing means 106 that works with the display processor 104 to suma preselected number of adjacent slices in each of the two images. Theoperator input means 100 is also connected with a single image slicestepping means 108, which can control the display processor 104 to stepthrough slices of only one of the two 3D images independently of theother.

Typically, 3-5 slices are fused together for display on monitor 22. Aglobal registration algorithm which registers the two three-dimensionalvolume images within ±2 slices is considered accurate registration dueto the fusing of the number of adjacent images.

Although described with particular reference to CT scanner imaging, itis to be appreciated that this technique is also applicable to magneticresonance images, PET images, SPECT images, and other three-dimensionaldiagnostic images. Moreover, the images being registered may be frommixed modalities. For example, a CT image can be registered using thistechnique with a PET image. When mixing modalities, care should be takento assure that the features are defined in both imaging modalities orappropriate adjustment made. Further, although described with particularreference to the lungs, it is to be appreciated that this technique isalso applicable to studies of other organs such as the colon, the liver,and other non-rigid organs. Moreover, this technique is also applicableto rigid portions of the body such as the head.

The invention has been described with reference to the preferredembodiments. Modifications and alterations will occur to others upon areading and understanding of the preceding detailed description. It isintended that the invention be construed as including all suchmodifications and alterations insofar as they come within the scope ofthe appended claims or the equivalents thereof.

1. An apparatus for diagnostic imaging comprising: a first memory meansfor storing a first diagnostic image; a second memory means for storinga second diagnostic image; a means for automatically registering thefirst and second diagnostic images from the first and second imagememories without operator assistance; a means for concurrentlydisplaying a corresponding pair of slices of the first and secondregistered diagnostic images; and a means for concurrently stepping thedisplayed slice pair corresponding through the first and secondregistered images.
 2. The apparatus as set forth in claim 1, wherein theregistering means includes: a means for determining an affine transformrepresentative of misalignment of the first and second diagnosticimages; and a means for operating on one of the first and seconddiagnostic images in accordance with the determined affine transform toregister the first and second images.
 3. The apparatus as set forth inclaim 2, wherein the affine transform determining means includes: ameans for matching pairs of points in the first and second diagnosticimages; a means for determining differences between locations andsurface normals of the matched points; and a means for determining anaffine transform which minimizes the deviation between the locations ofthe matched points.
 4. The apparatus as set forth in claim 3, whereinthe point matching means includes: a processing means which implements aK-D tree matching algorithm.
 5. The apparatus as set forth in claim 4,wherein the deviation minimizing means includes: a processor whichperforms a Levenberg-Marquardt error minimization algorithm.
 6. Theapparatus as set forth in claim 3, wherein the affine transform meansfurther includes: a means for selecting a reduced fraction of points tobe matched in the first and second registered images.
 7. The apparatusas set forth in claim 6, further including: a means for removing matchedpairs of points which fail to meet preselected criteria.
 8. Theapparatus as set forth in claim 3, further including: a selectivelynon-uniform random number of point reducing means, which reduces anumber of points by one of selectively using prior knowledge andrandomly while oversampling points for optimizing registration along adirection in which the stepping means steps the slice pairs.
 9. Theapparatus as set forth in claim 3, wherein the registering means furtherincludes: a means for converting the first and second diagnostic imagesinto feature image representations, the affine transform determiningmeans operating on the first and second features representations todetermine the affine transform.
 10. The apparatus a set forth in claim9, wherein the features generating means includes: a segmentation means,which segments appropriate target organs in the diagnostic images; and afeature extraction means that extracts a set of features to be matchedin the diagnostic images.
 11. The apparatus as set forth in claim 10,wherein the features image generating means includes: a thresholdingsegmentation means, which segments lungs in the diagnostic images usinga predetermined threshold and the features defined as the surface pointsof the lungs extracted by assigning a tissue on one side of a boundaryof an organ of interest a first value and a tissue or air on anotherside of the boundary of the organ of interest a second value, distinctfrom the first value; and a means for extracting a boundary layer ofvoxels of the first value which adjoin voxels of the second value. 12.The apparatus as set forth in claim 11, further including: a scalingmeans scaling the boundary layers of the two images; and a normalizingmeans for normalizing the boundary layers, prior to the surface imagesbeing operated on by the affine transform means.
 13. The apparatus asset forth in claim 9, wherein the affine transform determining meansfurther includes: a transform processors, which operates on one of thefeature images with the determined affine transform to facilitateiterative operation of the affine transform determining means tooptimize the affine transform.
 14. The apparatus as set forth in claim3, further including: a means for combining an operator selectedplurality of slices in each of the displayed slice images.
 15. Theapparatus as set forth in claim 3, further including: a diagnosticimaging apparatus connected with the first memory means for generatingthe first diagnostic image representation of a region of interest of apatient; and an archive means, from which the second imagerepresentation of the volume of interest of the patient taken at anearlier time is withdrawn and loaded into the second memory means.
 16. Amethod of diagnostic imaging comprising: storing a first diagnosticimage; storing a second diagnostic image; automatically registering thefirst and second diagnostic images without operator assistance;concurrently displaying a corresponding pair of slices of the first andsecond registered diagnostic images; and concurrently stepping thedisplayed slice pair to corresponding regions of the first and secondimages.
 17. The method as set forth in claim 16, wherein the step ofregistering includes: determining an affine transform representative ofmisalignment of the first and second diagnostic images; and operating onone of the first and second diagnostic images in accordance with thedetermined affine transform to register the first and second images. 18.The method as set forth in claim 17, wherein the step of determining theaffine transform includes: matching pairs of points in the first andsecond diagnostic images; determining differences between locations andsurface normals of the matched points; and minimizing the deviationbetween the locations of the matched points.
 19. The method as set forthin claim 18, wherein the step of matching includes: implementing a K-Dtree matching algorithm.
 20. The method as set forth in claim 19,wherein the deviation minimizing step includes: utilizing aLevenberg-Marquardt error minimization algorithm.
 21. The method as setforth in claim 18, wherein the step of determining the affine transformfurther includes: selecting a reduced fraction of points to be matchedin the two images.
 22. The method as set forth in claim 21, furtherincluding: removing matched pairs of points that fail to meetpreselected criteria.
 23. The method as set forth in claim 18, furtherincluding: reducing a number of points selectively, non-uniformly by oneof prior knowledge and randomly with an oversampling of points foroptimizing registration along a direction in which the slice pairs arestepped.
 24. The method as set forth in claim 18, wherein: the step ofregistering further includes converting the first and second diagnosticimages into feature image representations; and the step of determiningthe affine transform further includes operating on the first and secondfeature image representations to determine the affine transform.
 25. Themethod as set forth in claim 24 wherein generating the feature imageincludes: segmenting target organs in the diagnostic images; andextracting a set of features to be matched in the diagnostic image. 26.The method as set forth in claim 25 wherein generating the feature imageincludes: segmenting lungs in the diagnostic images to assign tissue onone side of a boundary of an organ of interest a first value and tissueor air on another side of the boundary of the organ of interest a secondvalue, distinct from the first value; and extracting a boundary layer ofvoxels of the organ of interest.
 27. The method as set forth in claim26, further including prior to determining the affine transform: scalingthe boundary layer; and normalizing the boundary layer.
 28. The methodas set forth in claim 26, further including: operating on one of theboundary layers with the determined affine transform; and iterativelydetermining correction transforms to the affine transform to optimizethe affine transform.
 29. The method as set forth in claim 26, furtherincluding: combining an operator selected plurality of slices in each ofthe displayed slice images.
 30. The method as set forth in claim 16,further including: generating a current diagnostic image representationof a region of interest of a patient; and retrieving a previouslygenerated image representation of the volume of interest of the patient.